Hadron spectra and elliptic flow in relativistic heavy-ion collisions are studied in event-by-event (3+1)D ideal hydrodynamic simulations with fluctuating initial conditions given by the AMPT Monte Carlo model. Both the coherent soft gluon production from wounded nucleons and incoherent mini-jet production in semi-hard parton scatterings are considered. These partons take part in parton cascade and are assumed to be locally thermalized. They provide the fluctuating initial conditions for hydrodynamic simulations with a Gaussian smearing. Effects of both transverse and longitudinal fluctuations are studied. The initial fluctuations along rapidity direction lead to expanding hot spots in longitudinal direction, which will reduce elliptic flow and the yield of particles at high transverse momentum. The intrinsic correlation introduced in the fluctuating initial conditions is also found to influence the di-hadron correlation of the final hadrons.